Extracting Comprehensible Concept Representations from Trained Neural Networks

Although they are applicable to a wide array of problems, and have demonstrated good performance on a number of diicult, real-world tasks, neural networks are not usually applied to problems in which compre-hensibility of the acquired concepts is important. The concept representations formed by neural networks are hard to understand because they typically involve distributed, nonlinear relationships encoded by a large number of real-valued parameters. To address this limitation, we have been developing algorithms for extracting \symbolic" concept representations from trained neural networks. We rst discuss why it is important to be able to understand the concept representations formed by neural networks. We then brieey describe our approach and discuss a number of issues pertaining to comprehensibility that have arisen in our work. Finally, we discuss choices that we have made in our research to date, and open research issues that we have not yet addressed.